Skip to content

[SDTEST-3825] Treat disabled managed tests as skipped#74

Merged
anmarchenko merged 2 commits into
mainfrom
anmarchenko/disabled-test-management-skips
Jun 5, 2026
Merged

[SDTEST-3825] Treat disabled managed tests as skipped#74
anmarchenko merged 2 commits into
mainfrom
anmarchenko/disabled-test-management-skips

Conversation

@anmarchenko
Copy link
Copy Markdown
Member

@anmarchenko anmarchenko commented Jun 4, 2026 •
edited
Loading

What

Treat disabled tests from Datadog Test Management as skipped during planning, alongside TIA skippable tests. Module is now part of test FQNs for both TIA skippable data and test-management disabled data so same suite/name tests in different modules do not collide.

Why

Test Management can mark tests as disabled, and ddtest should honor that state when deciding which discovered tests to skip. This keeps planner behavior aligned with Datadog-managed test state while preserving the existing rule that full test discovery only depends on TIA/test-skipping settings.

E2E testing

  1. In a repository with TIA and Test Management enabled, mark one test as disabled in Datadog Test Management and run ddtest plan; confirm the generated plan skips the test file that contains only disabled tests even when they are not returned as TIA-skippable.

  2. Check that skipping test files where tests are skipped by TIA still work.

  3. Check that test files where some tests are disabled and some are skipped by TIA is also fully skipped by ddtest planning

@anmarchenko anmarchenko marked this pull request as ready for review June 4, 2026 15:12
@anmarchenko anmarchenko requested a review from a team as a code owner June 4, 2026 15:12
chatgpt-codex-connector[bot]
chatgpt-codex-connector Bot reviewed Jun 4, 2026
View reviewed changes
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1b9616c134

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread internal/testoptimization/test_management.go
@anmarchenko anmarchenko changed the title Treat disabled managed tests as skipped [SDTEST-3825] Treat disabled managed tests as skipped Jun 5, 2026
@anmarchenko
Copy link
Copy Markdown
Member Author

anmarchenko commented Jun 5, 2026

E2E Test Report: SUCCESS

Tested by: Shepherd Agent (autonomous QA for Datadog Test Optimization)

Test Environment

  • Method: Local Shepherd run with sidekiq playground, ddtest-plan, and local Mockdog
  • Revision tested: 1a69baa2630059039877d3e54c24cb348812bea8
  • Branch tested: anmarchenko/disabled-test-management-skips
  • Command shape: bin/crook run sidekiq -c ddtest-plan --dep ddtest=anmarchenko/disabled-test-management-skips --scenario ddtest-sidekiq-tm-disabled-tia-mix --debug

Results

Check Status Evidence
Disabled-only Test Management file skipped PASS test/systemd_test.rb had its only test disabled and was absent from runner/test-files.txt and every runner-* split
TIA-only file skipped PASS test/sharding_test.rb had both tests returned by /api/v2/ci/tests/skippable and was absent from runner/test-files.txt and every runner-* split
Mixed TIA + disabled file skipped PASS test/dead_set_test.rb had 1 TIA-skippable test plus 2 disabled tests and was absent from runner/test-files.txt and every runner-* split
Module-aware backend data exercised PASS Mockdog returned skippable entries with module: "minitest"; Test Management response grouped disabled tests under the minitest module

Planner Evidence

  • tiaSkippableTestsCount=3
  • disabledTestsCount=3
  • skippableTestsCount=6
  • Plan report: Fully skipped files: 3, Test files to run: 38
  • Runner split: 4 runners, with 38 total planned files

Issues Found

Shepherd Mockdog did not previously support module on skippable_tests, which this PR now needs for module-aware TIA matching. I updated the local Shepherd test backend to support module-aware skippable responses and used that path for this E2E run. No issue was found in the ddtest PR behavior under test.

Verification

This was a planning-only local Mockdog E2E run, so no live Datadog UI verification was performed. The verification target was the generated ddtest plan and cached backend responses.

This E2E test was performed by Shepherd - autonomous QA agent for Datadog Test Optimization

@anmarchenko anmarchenko merged commit 4f27bad into main Jun 5, 2026
3 checks passed
@anmarchenko anmarchenko deleted the anmarchenko/disabled-test-management-skips branch June 5, 2026 08:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@chatgpt-codex-connector chatgpt-codex-connector[bot] chatgpt-codex-connector[bot] left review comments

@juan-fernandez juan-fernandez juan-fernandez approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@anmarchenko @juan-fernandez